Procesamiento de Datos Heterogéneos en el Internet de las Cosas

Day after day the number of Internet of Things (IoT) and smart devices capable of producing, consuming, and exchanging information increases considerably. In most cases, the structure of the information produced by such devices is completely different, therefore providing heterogeneous information. This fact is becoming a challenge for researchers working on IoT, who need to perform homogenisation and pre-processing tasks before using the IoT data in their analytics. Moreover, the volume of these heterogeneous data sources is usually huge, thus leading us to the Big Data term, which relies on the three V’s: Velocity, Volume, and Variety. Being able to work with these large and heterogeneous datasets, performing specific domain analytics, and reacting in real time to situations of interests, would result in a big competitive advantage. Hence, there is a need of being able to operate with these heterogeneous data, to consume, to process, and to analyse them. In this context, Data Serialization Systems (DSS), Stream Processing (SP) platforms, and Complex Event Processing (CEP) are postulated as potential tools that will help developers to overcome these challenges previously commented. Firstly, DSS allows us to transmit and transport data quickly and effectively thanks to their serialization strategies. Secondly, SP platforms bring the possibility of establishing architectures capable of consuming, processing, and transforming vast amounts of data in real time. Finally, CEP is a well-established technology that facilitates the analytics of streams of data, detecting and notifying about anomalies in real time. At the same time, these advantageous tools require years of training to be able to dominate and use them efficiently and effectively. So, providing these technologies to domain experts, users who are experts on the domain itself but usually lack computer science or programming skills, is a must. This is where Model-Driven Development (MDD) comes up. MDD is a paradigm in software development that facilitates users the usage of complex technologies, due to it abstracts the user from the implementation details and allows them to focus on defining the problem directly. Therefore, in this PhD thesis, we aim to solve these issues. On the first hand, we have developed an architecture for processing and analysing data coming from heterogeneous sources with different structures in IoT scopes, allowing researchers to focus on data analysis, without having to worry about the structures of the data that are going to be processed. This architecture combines the real-time SP paradigm and DSS for information processing and transforming, together with the CEP for information analysis. The combination of these three technologies allows developers and researchers to build systems that can consume, process, transform, and analyse large amounts of heterogeneous data in real time. On the other hand, to bring this architecture to any kind of users, we have developed MEdit4CEP-SP, a model-driven system and extension of the tool MEdit4CEP, that integrates SP, DSS, and CEP for consuming, processing and analysing heterogeneous data in real time, providing domain experts with a graphical editor which allows them to infer and define heterogeneous data domains, as well as user-friendly modelling the situations of interest to be detected in such domains. In this editor, the graphical definitions are automatically transformed into code, thanks to the use of MDD techniques, which is deployed in the processing system at runtime. Also, all these definitions are persistently stored in a NoSQL database, so any user can reuse the definitions that already stored. Moreover, this set of tools could be used as collaborative, due to they can be deployed on the cloud, meaning that several domain experts or final users can be working together with their MEdit4CEP-SP instances, using their own computers, adding, removing and updating event types and event patterns from the same CEP engine. Furthermore, we have evaluated our solution thoroughly. First, we have tested our SP architecture to prove its scalability and its computing capacity, showing that the system can process more data using more nodes. Its performance is outstanding, reaching a maximum Transactions Per Second (TPS) of 135 080 Mb/s using 4 nodes. Next, we have tested the graphical editor with real users to show that it provides its functionalities in a friendly and intuitive way. The users were asked to fulfil a series of tasks and then they answered a questionnaire to evaluate their experience with the editor. The results of such questionnaire were successful. Finally, the benefits of this system are compared with other existing approaches in the literature with excellent results. In such comparative analysis, we have contrasted our proposal against others based on a series of key features that systems for modelling, consuming, processing, and analysing heterogeneous data in real time should present.Hoy en día, el número de dispositivos del Internet de las Cosas (Internet of Things, IoT) y dispositivos inteligentes capaces de producir, consumir e intercambiar información crece considerablemente. En la mayoría de casos, la estructura de la información producida por dichos dispositivos es completamente diferente, produciendo pues información heterogénea. Este hecho se está convirtiendo en todo un desafío para investigadores que trabajan en IoT, que necesitan llevar a cabo tareas de pre-procesamientos y homogeneización antes de poder hacer uso de estos datos del IoT en sus análisis. Además, el volumen de estas fuentes de datos heterogéneas es normalmente desmesurado, lo cual nos lleva al término del Big Data que se basa en las tres Vs: Velocidad, Volumen y Variedad. Ser capaces de trabajar con estos inmensos y heterogéneos volúmenes de datos, realizando análisis de dominio específico, y reaccionando a determinadas situaciones de interés en tiempo real, resultaría en una enorme ventaja competitiva. Por lo tanto, existe la necesidad de ser capaces de operar con estos datos heterogéneos, consumirlos, procesarlos y analizarlos. En este contexto, los Sistemas de Serialización de Datos (Data Serialization Systems, DSS), las plataformas de Procesamiento de Flujos (Stream Processing, SP) de datos y el Procesamiento de Eventos Complejos (Complex Event Processing, CEP) se han postulado como poderosas herramientas que ayudarán a los desarrolladores a superar estos desafíos previamente comentados. En primer lugar, DSS nos permite transmitir y transportar datos de una forma rápida y efectiva gracias a sus estrategias de serialización. En segundo lugar, las plataformas SP brindan la posibilidad de establecer arquitecturas capaces de consumir, procesar y transformar ingentes cantidades de datos en tiempo real. Finalmente, CEP es una tecnología bien conocida que facilita el análisis de flujos de datos, detectando y notificando anomalías en tiempo real. Al mismo tiempo, estas ventajosas herramientas requieren años de entrenamiento para poder ser dominadas y usadas de forma eficiente y efectiva. De esa forma, proporcionar estas tecnologías a los expertos del dominio, usuarios que son expertos en el dominio a tratar pero que, normalmente, carecen de habilidades de informática o programación, es una obligación. Aquí es donde el Desarrollo Dirigido por Modelos (Model-Driven Development, MDD) entra en escena. MDD es un paradigma en el desarrollo de software que facilita a los usuarios la utilización de tecnologías complejas ya que abstrae al usuario de los detalles de implementación y les permite centrarse en definir el problema directamente. Así pues, en esta Tesis Doctoral, perseguimos resolver estos problemas. Por un lado, hemos desarrollado una arquitectura para procesar y analizar datos provenientes de fuentes heterogéneas con diferentes estructuras en ámbitos del IoT, permitiendo a los investigadores centrarse en el análisis de los datos, sin tener que preocuparse por la estructura de los datos que van a ser procesados. Esta arquitectura combina el paradigma de SP y DSS, para procesar y transformar la información, junto a CEP para analizar dichos datos, todo ello en tiempo real. La combinación de estas tres tecnologías permite a los desarrolladores e investigadores construir sistemas capaces de consumir, procesar, transformar y analizar grandes cantidades de datos heterogéneos en tiempo real. Por otro lado, con la idea de poder acercar esta arquitectura a cualquier tipo de usuario, a partir de ella hemos desarrollado MEdit4CEP-SP, un sistema dirigido por modelos y una extensión de MEdit4CEP, que integra SP, DSS y CEP para consumir, procesar y analizar datos heterogéneos en tiempo real, proporcionándole a los expertos del dominio un editor grafico que les permita inferir y definir los dominios de datos heterogéneos, así como poder modelar de una forma amigable las diferentes situaciones de interés que se persiguen detectar en dichos dominios. En este editor, las definiciones gráficas son automáticamente transformadas en código, gracias al uso de técnicas de MDD, que es desplegado en el sistema de procesamiento en tiempo de ejecución. Además, todas estas definiciones son almacenadas persistentemente en una base de datos NoSQL, de forma que cualquier usuario puede reutilizar las definiciones que ya han sido almacenadas. Este conjunto de herramientas puede ser utilizado de forma colaborativa, ya que pueden ser desplegadas en la nube, de forma que diferentes expertos del dominio o usuarios finales pueden estar trabajando juntos con sus propias instancias de MEdit4CEP-SP en sus ordenadores, añadiendo, eliminando y actualizando tipos de eventos y patrones sobre el mismo motor CEP. Por otro lado, hemos evaluado nuestra solución exhaustivamente. Primero, hemos probado nuestra arquitectura con el objetivo de demostrar su escalabilidad y capacidad de computación, mostrando que el sistema es capaz de procesar más datos utilizando más nodos. Su rendimiento es excepcional, alcanzando en Transacciones por Segundo (Transanctions Per Second, TPS) un máximo de 135 080 Mb/s utilizando 4 nodos. A continuación, hemos probado el editor gráfico con usuarios reales para mostrar que este proporciona sus funcionalidades de una forma amigable e intuitiva. Los usuarios fueron sometidos a una serie de tareas y después tuvieron que contestar a un cuestionario que evaluaba su experiencia con el editor. Los resultados de dicho cuestionario fueron exitosos. Finalmente, los beneficios de este sistema han sido comparados con otros enfoques existentes en la literatura con excelentes resultados. En dicho análisis comparativo, hemos contrastado nuestra propuesta contra las demás utilizando una serie de requisitos indispensables que los sistemas para modelar, consumir, procesar y analizar datos heterogéneos en tiempo real deberían proporcionar.Universidad de Cádiz (2017-020/PU/EPIF-FPI-CT/CP) Ministerio de Economía y Competitividad (TIN2015-65845-C3-3-R) Ministerio De Ciencia, Innovación y Universidades (RTI2018-093608-B-C33

Sistema en la nube para la monitorización y alerta de la calidad del aire en tiempo real

En este TFG hemos desarrollado un sistema en la nube capaz de detectar, en tiempo real, una serie de eventos complejos relativos a la calidad del aire; el sistema, posteriormente, notifica a una serie de usuarios finales la calidad del aire en cada instante mediante el uso de una aplicación Android. Las notificaciones que estos usuarios recibirán estarán adaptadas a su contexto, es decir, ubicación y el tipo de actividad que se encuentren realizando, así como a sus circunstancias personales (edad, enfermedades respiratorias, etc.)Número de páginas: 186

MEdit4CEP-SP: A model-driven solution to improve decision-making through user-friendly management and real-time processing of heterogeneous data streams

Organisations today are constantly consuming and processing huge amounts of data. Such datasets are often heterogeneous, making it difficult to work with them quickly and easily due to their format constraints or their disparate data structures. Therefore, being able to efficiently and intuitively work with such data to analyse them in real time to detect situations of interest as quickly as possible is a great competitive advantage for companies. Existing approaches have tried to address this issue by providing users with analytics or modelling tools in an isolated way, but not combining them as a onein- all solution. In order to fill this gap, we present MEdit4CEP-SP, a model-driven system that integrates Stream Processing (SP) and Complex Event Processing (CEP) technologies for consuming, processing and analysing heterogeneous data in real time. It provides domain experts with a graphical editor that allows them to infer and define heterogeneous data domains, while also modelling, in a user-friendly way, the situations of interest to be detected in such domains. These graphical definitions are then automatically transformed into code, which is deployed in the processing system at runtime. The alerts detected by the system, in real-time, allow users to react as quickly as possible, thus improving the decision-making process. Additionally, MEdit4CEP-SP provides persistence, storing these definitions in a NoSQL database to permit their reuse by other instances of the system. Further benefits of this system are evaluated and compared with other existing approaches in this paper.Las organizaciones hoy en día están constantemente consumiendo y procesando grandes cantidades de datos. Tales conjuntos de datos son a menudo heterogéneos, lo que dificulta trabajar con ellos de manera rápida y fácil debido a sus restricciones de formato o sus estructuras de datos dispares. Por lo tanto, ser capaz de trabajar de manera eficiente e intuitiva con estos datos para analizarlos en tiempo real para detectar situaciones de interés lo más rápido posible es una gran ventaja competitiva para las empresas. Los enfoques existentes han intentado abordar este problema proporcionando a los usuarios herramientas de análisis o modelado de forma aislada, pero no combinándolas como una solución todo en uno. Para llenar este vacío, presentamos MEdit4CEP-SP, un sistema impulsado por modelos que integra las tecnologías de Procesamiento de Flujo (SP) y Procesamiento de Eventos Complejos (CEP) para consumir, procesar y analizar datos heterogéneos en tiempo real. Proporciona a los expertos del dominio un editor gráfico que les permite inferir y definir dominios de datos heterogéneos, al mismo tiempo que modela, de una manera amigable para el usuario, las situaciones de interés a ser detectadas en dichos dominios. Estas definiciones gráficas se transforman automáticamente en código, que se despliega en el sistema de procesamiento en tiempo de ejecución. Las alertas detectadas por el sistema, en tiempo real, permiten a los usuarios reaccionar lo más rápido posible, mejorando así el proceso de toma de decisiones. Además, MEdit4CEP-SP proporciona persistencia, almacenando estas definiciones en una base de datos NoSQL para permitir su reutilización por otras instancias del sistema. Los beneficios adicionales de este sistema se evalúan y comparan con otros enfoques existentes en este documento.This work was partly supported by the Spanish Ministry of Science and Innovation and the European Regional Development Fund (ERDF) under project FAME (RTI2018-093608-B-C33), and also by the pre-doctoral program of the University of Cádiz, Spain (2017-020/PU/EPIF-FPI-CT/CP)

A stream processing architecture for heterogeneous data sources in the Internet of Things

The number of Internet of Things (IoT) and smart devices capable of producing, consuming and exchanging information is constantly increasing. It is estimated there will be around 30 billion of them in 2020. In most cases, the structures of the information produced by such devices are completely different, thus providing heterogeneous information. This is becoming a challenge for researchers working on IoT, who need to perform homogenisation and pre-processing tasks before using the IoT data. This paper aims to provide an architecture for processing and analysing data from heterogeneous sources with different structures in IoT scopes, allowing researchers to focus on data analysis, without having to worry about the structure of the data sources. This architecture combines the real-time stream processing paradigm for information processing and transforming, together with the complex event processing for information analysis. This provides us with capability of processing, transforming and analysing large amounts of information in real time. The results obtained from the evaluation of a real-world case study about water supply network management show that the architecture can be applied to an IoT water management scenario to analyse the information in real time. Additionally, the stress tests successfully conducted for this architecture highlight that a large incoming rate of input events could be processed without latency, resulting in efficient performance of the proposed architecture. This novel software architecture is adequate for automatically detecting situations of interest in the IoT through the processing, transformation and analysis of large amounts of heterogeneous information in real time.El número de dispositivos del Internet de las Cosas (IoT) y dispositivos inteligentes capaces de producir, consumir e intercambiar información está aumentando constantemente. Se estima que habrá alrededor de 30 mil millones de ellos en 2020. En la mayoría de los casos, las estructuras de la información producida por dichos dispositivos son completamente diferentes, proporcionando así información heterogénea. Esto se está convirtiendo en un desafío para los investigadores que trabajan en IoT, quienes necesitan realizar tareas de homogeneización y preprocesamiento antes de utilizar los datos de IoT. Este documento tiene como objetivo proporcionar una arquitectura para procesar y analizar datos de fuentes heterogéneas con diferentes estructuras en ámbitos de IoT, permitiendo a los investigadores centrarse en el análisis de datos, sin tener que preocuparse por la estructura de las fuentes de datos. Esta arquitectura combina el paradigma de procesamiento de flujo en tiempo real para el procesamiento y transformación de información, junto con el procesamiento de eventos complejos para el análisis de información. Esto nos proporciona la capacidad de procesar, transformar y analizar grandes cantidades de información en tiempo real. Los resultados obtenidos de la evaluación de un estudio de caso del mundo real sobre la gestión de la red de suministro de agua muestran que la arquitectura puede ser aplicada a un escenario de gestión de agua de IoT para analizar la información en tiempo real. Además, las pruebas de estrés realizadas con éxito para esta arquitectura destacan que una gran tasa de entrada de eventos de entrada podría ser procesada sin latencia, lo que resulta en un rendimiento eficiente de la arquitectura propuesta. Esta novedosa arquitectura de software es adecuada para detectar automáticamente situaciones de interés en el IoT a través del procesamiento, transformación y análisis de grandes cantidades de información heterogénea en tiempo real.This work was supported in part by the Spanish Ministry of Science and Innovation and the European Union FEDER Funds (No. TIN2015-65845-C3-3-R and No. RTI2018-093608-B-C33) and in part by the pre-doctoral program of the University of Cádiz (2017-020/PU/EPIF-FPI-CT/CP). In addition, we would like to thank GEN Grupo Energético for sharing their data for testing purposes

An Internet of Things Platform for Air Station Remote Sensing and Smart Monitoring

Air pollution is currently receiving more attention by international governments and organizations. Nevertheless, current systems for air quality monitoring lack essential requirements which are key in order to be effective concerning users’ access to the information and efficient regarding real-time monitoring and notification. This paper presents an Internet of Things platform for air station remote sensing and smart monitoring that combines Big Data and Cloud Computing paradigms to process and correlate air pollutant concentrations coming from multiple remote stations, as well as to trigger automatic and personalized alerts when a health risk for their particular context is detected. This platform has been tested by analyzing the results of observing Andalusian, South of Spain, sensor network during a long period of time. The results show that this novel solution can help to reduce the impact of air pollution on human health since citizens are alerted in real time

A microservice architecture for real-time IoT data processing: A reusable Web of things approach for smart ports

Major advances in telecommunications and the Internet of Things have given rise to numerous smart city scenarios in which smart services are provided. What was once a dream for the future has now become reality. However, the need to provide these smart services quickly, efficiently, in an interoperable manner and in real time is a cutting-edge technological challenge. Although some software architectures offer solutions in this area, these are often limited in terms of reusability and maintenance by independent modules —involving the need for system downtime when maintaining or evolving, as well as by a lack of standards in terms of the interoperability of their interface. In this paper, we propose a fully reusable microservice architecture, standardized through the use of the Web of things paradigm, and with high efficiency in real-time data processing, supported by complex event processing techniques. To illustrate the proposal, we present a fully reusable implementation of the microservices necessary for the deployment of the architecture in the field of air quality monitoring and alerting in smart ports. The performance evaluation of this architecture shows excellent results

Un Recorrido por los Principales Proveedores de Servicios de Machine Learning y Predicción en la Nube

Los medios tecnológicos para el consumo, producción e intercambio de información no hacen más que aumentar cada día que pasa. Nos encontramos envueltos en el fenómeno Big Data, donde ser capaces de analizar esta informa ción con el objetivo de poder inferir situaciones del futuro basándonos en datos del pasado y del presente, nos puede reportar una ventaja competitiva que nos distinga claramente de otras opciones. Dentro de las múltiples disciplinas exis tentes para el análisis de grandes cantidades información encontramos el Ma chine Learning y, a su vez, dentro de este podemos destacar la capacidad predic tiva que nos proporcionan muchas de las opciones existentes actualmente en el mercado. En este trabajo realizamos un análisis de estas principales opciones de APIs predictivas en la nube, las comparamos entre sí, y finalmente llevamos a cabo una experimentación con datos reales de la Red de Vigilancia y Control de la Calidad del Aire de la Junta de Andalucía. Los resultados demuestran que estas herramientas son una opción muy interesante a considerar a la hora de tratar de predecir valores de contaminantes que pueden afectar a nuestra salud seriamente, pudiéndose llevar a cabo acciones preventivas sobre la población afectadaMinisterio de Economía y Competitividad TIN2015-65845-C3-3-RMinisterio de Economía y Competitividad TIN2016-81978-RED

Air4People: a Smart Air Quality Monitoring and Context-Aware Notification System

Over the last years, air pollution and air quality have received increasing attention in the scope of Internet of Things and smart cities, since they can seriously affect citizens' health. However, current systems for air quality monitoring and notification lack essential key requirements in order to be effective as far as users' access to the information is concerned and, particularly, the provision of context-aware notifications. This paper presents Air4People, an air quality monitoring and context-aware notification system, which submits personalized alerts to citizens based on several types of context, whenever air quality-related health risks are detected for their particular context

A microservice architecture for real-time IoT data processing: A reusable Web of things approach for smart ports

Major advances in telecommunications and the Internet of Things have given rise to numerous smart city scenarios in which smart services are provided. What was once a dream for the future has now become reality. However, the need to provide these smart services quickly, efficiently, in an interoperable manner and in real time is a cutting-edge technological challenge. Although some software architectures offer solutions in this area, these are often limited in terms of reusability and maintenance by independent modules —involving the need for system downtime when maintaining or evolving, as well as by a lack of standards in terms of the interoperability of their interface. In this paper, we propose a fully reusable microservice architecture, standardized through the use of the Web of things paradigm, and with high efficiency in real-time data processing, supported by complex event processing techniques. To illustrate the proposal, we present a fully reusable implementation of the microservices necessary for the deployment of the architecture in the field of air quality monitoring and alerting in smart ports. The performance evaluation of this architecture shows excellent results. © 2021 The Author(s)This work was supported by the Spanish Ministry of Science and Innovation and the European Regional Development Fund (ERDF) under research projects CoSmart and FAME (ref. TIN2017–83964-R and RTI2018–093608-B-C33), and by regional projects (ref. CEIJ-C01.1 and CEIJ-C01.2) coordinated from UAL-UCA and funded by Campus de Excelencia Internacional del Mar (CEIMAR) consortium. We would like to thank J. M. Pérez Sánchez, A. M. Garrido López and R. J. Catalán Alonso from the Autoridad Portuaria de la Bahía de Cádiz, Spain, for sharing valuable knowledge on port management and environment and collaborating with us in this work

Association of mechanical bowel preparation with oral antibiotics and anastomotic leak following left sided colorectal resection: an international, multi-centre, prospective audit.

This is the peer reviewed version of the following article: , (2018), Association of mechanical bowel preparation with oral antibiotics and anastomotic leak following left sided colorectal resection: an international, multi‐centre, prospective audit. Colorectal Dis, 20: 15-32. doi:10.1111/codi.14362, which has been published in final form at https://doi.org/10.1111/codi.14362. This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Use of Self-Archived VersionsINTRODUCTION: The optimal bowel preparation strategy to minimise the risk of anastomotic leak is yet to be determined. This study aimed to determine whether oral antibiotics combined with mechanical bowel preparation (MBP+Abx) was associated with a reduced risk of anastomotic leak when compared to mechanical bowel preparation alone (MBP) or no bowel preparation (NBP). METHODS: A pre-planned analysis of the European Society of Coloproctology (ESCP) 2017 Left Sided Colorectal Resection audit was performed. Patients undergoing elective left sided colonic or rectal resection with primary anastomosis between 1 January 2017 and 15 March 2017 by any operative approach were included. The primary outcome measure was anastomotic leak. RESULTS: Of 3676 patients across 343 centres in 47 countries, 618 (16.8%) received MBP+ABx, 1945 MBP (52.9%) and 1099 patients NBP (29.9%). Patients undergoing MBP+ABx had the lowest overall rate of anastomotic leak (6.1%, 9.2%, 8.7% respectively) in unadjusted analysis. After case-mix adjustment using a mixed-effects multivariable regression model, MBP+Abx was associated with a lower risk of anastomotic leak (OR 0.52, 0.30-0.92, P = 0.02) but MBP was not (OR 0.92, 0.63-1.36, P = 0.69) compared to NBP. CONCLUSION: This non-randomised study adds 'real-world', contemporaneous, and prospective evidence of the beneficial effects of combined mechanical bowel preparation and oral antibiotics in the prevention of anastomotic leak following left sided colorectal resection across diverse settings. We have also demonstrated limited uptake of this strategy in current international colorectal practice